NYC Schools Perceptions

Based on responses to surveys designed to gauge parent, student, and teacher perceptions of the quality of New York City schools, we’ll going to investigate the following questions:

• Do student, teacher, and parent perceptions of NYC school quality appear to be related to demographic and academic success metrics?

• Do students, teachers, and parents have similar perceptions of NYC school quality?

The next two files, masterfile11_gened_final.xlsx and masterfile11_gened_final.txt, contain survey data for “general education” schools — those that do not specifically serve populations with special needs.

The files masterfile11_d75_final.xlsx and masterfile11_d75_final.txt contain survey data for District 75 schools, which provide special education support for children with special needs such as learning or physical disabilities.

you can download these files from: https://data.cityofnewyork.us/Education/2011-NYC-School-Survey/mnz3-dyi8

We’re going to use the combined data set that have the demographic data: “combined.csv” you can download the file from: https://data.world/dataquest/nyc-schools-data/workspace/file?filename=combined.csv

1. Load the packages we ’ll goint to use:

library(readr)
library(dplyr)

replacing previous import by ‘rlang::dots_n’ when loading ‘dplyr’
Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

library(stringr)
library(purrr)
library(tidyr)
library(ggplot2)

2. Read the csv and tsv files:

combined <- read_csv("/home/matias/Documents/repos/Data-Science-Notebooks/NYC Schools Perceptions/combined.csv")

Parsed with column specification:
cols(
  .default = col_double(),
  DBN = col_character(),
  school_name = col_character(),
  boro = col_character()
)
See spec(...) for full column specifications.

survey <- read_tsv("/home/matias/Documents/repos/Data-Science-Notebooks/NYC Schools Perceptions/2011 data files online/masterfile11_gened_final.txt")

Parsed with column specification:
cols(
  .default = col_double(),
  dbn = col_character(),
  bn = col_character(),
  schoolname = col_character(),
  studentssurveyed = col_character(),
  schooltype = col_character(),
  p_q1 = col_logical(),
  p_q3d = col_logical(),
  p_q9 = col_logical(),
  p_q10 = col_logical(),
  p_q12aa = col_logical(),
  p_q12ab = col_logical(),
  p_q12ac = col_logical(),
  p_q12ad = col_logical(),
  p_q12ba = col_logical(),
  p_q12bb = col_logical(),
  p_q12bc = col_logical(),
  p_q12bd = col_logical(),
  t_q6m = col_logical(),
  t_q9 = col_logical(),
  t_q10a = col_logical()
  # ... with 18 more columns
)
See spec(...) for full column specifications.

survey_d75 <- read_tsv("/home/matias/Documents/repos/Data-Science-Notebooks/NYC Schools Perceptions/2011 data files online/masterfile11_d75_final.txt")

Parsed with column specification:
cols(
  .default = col_double(),
  dbn = col_character(),
  bn = col_character(),
  schoolname = col_character(),
  studentssurveyed = col_character(),
  schooltype = col_character(),
  p_q5 = col_logical(),
  p_q9 = col_logical(),
  p_q13a = col_logical(),
  p_q13b = col_logical(),
  p_q13c = col_logical(),
  p_q13d = col_logical(),
  p_q14a = col_logical(),
  p_q14b = col_logical(),
  p_q14c = col_logical(),
  p_q14d = col_logical(),
  t_q11a = col_logical(),
  t_q11b = col_logical(),
  t_q14 = col_logical(),
  t_q15a = col_logical(),
  t_q15b = col_logical()
  # ... with 14 more columns
)
See spec(...) for full column specifications.

combined

survey

survey_d75

Simplifying the survey data frames to include only variables we will need for our analysis:

combined_up <-combined %>%
  select(1, 3:7, 13, 15, 19:22, 27:30)
  
  
survey_up <- survey %>%
  filter((schooltype == "High School") | (schooltype == "Middle / High School")) %>%
  select(1, 17:32)
  
survey_d75_up <- survey_d75 %>%
  select(1, 17:32)
  
combined_up

survey_up

survey_d75_up

Combining survey data frames first:

We can use dplyr bind_rows() to combine data frames with differents number of columns, but because both data frames have the same number of it, we can use simple rbind.

combined_surveys <- rbind(survey_up, survey_d75_up)
combined_up <- combined_up %>%
  rename(dbn = `DBN`)

Joining combined survey data frame with combined_up data frame (demographic data):

combined_final <- combined_up %>%
  left_join(combined_surveys, by = "dbn")
combined_final <- combined_final %>%
  mutate(quality_metrics = (`saf_tot_11`+`com_tot_11`+`eng_tot_11`+`aca_tot_11`)/4)

Let’s get an idea of which demographic and test score variables may be related to parent with a correlation matrix:

cor_mat <- combined_final %>%
  select_if(is.numeric) %>% 
  cor(use = "pairwise.complete.obs")

Subsetting the correlation matrix:

cor_tib <- cor_mat %>%
  as_tibble(rownames = "variable")
 
saf_tot_11_cors <- cor_tib %>%
  filter((variable == "avg_sat_score") | (variable == "asian_per") | (variable == "white_per") | (variable == "black_per") | (variable == "hispanic_per") | (variable == "lat") | (variable == "long") | (variable == "dropout_percent")) %>%
  select(variable, saf_tot_11) %>%
  filter(saf_tot_11 > 0.15 | saf_tot_11 < -0.15)
com_tot_11_cors <- cor_tib %>%
  filter((variable == "avg_sat_score") | (variable == "asian_per") | (variable == "white_per") | (variable == "black_per") | (variable == "hispanic_per") | (variable == "lat") | (variable == "long") | (variable == "dropout_percent")) %>%
  select(variable, com_tot_11) %>%
  filter(com_tot_11 > 0.15 | com_tot_11 < -0.15)
eng_tot_11_cors <- cor_tib %>%
  filter((variable == "avg_sat_score") | (variable == "asian_per") | (variable == "white_per") | (variable == "black_per") | (variable == "hispanic_per") | (variable == "lat") | (variable == "long") | (variable == "dropout_percent")) %>%
  select(variable, eng_tot_11) %>%
  filter(eng_tot_11 > 0.15 | eng_tot_11 < -0.15)
aca_tot_11_cors <- cor_tib %>%
  filter((variable == "avg_sat_score") | (variable == "asian_per") | (variable == "white_per") | (variable == "black_per") | (variable == "hispanic_per") | (variable == "lat") | (variable == "long") | (variable == "dropout_percent")) %>%
  select(variable, aca_tot_11) %>%
  filter(aca_tot_11 > 0.15 | aca_tot_11 < -0.15)
quality_metrics_cors <- cor_tib %>%
  filter((variable == "avg_sat_score") | (variable == "asian_per") | (variable == "white_per") | (variable == "black_per") | (variable == "hispanic_per") | (variable == "lat") | (variable == "long") | (variable == "dropout_percent")) %>%
  select(variable, quality_metrics) %>%
  filter(quality_metrics > 0.15 | quality_metrics < -0.15)

Scatter plots with the variables filtered:

A group for scatter plots for each survey variable: saf_tot_11, com_tot_11, eng_tot_11, aca_tot_11, quality_metrics (avg of saf, com, eng, aca).

First: “Quality metrics vs Demographic Variables” - colour: Deepskyblue

create_scatter <- function(x,y) {
  ggplot(data = combined_final) + 
     aes_string(x = x, y = y) +  
     geom_point(na.rm=TRUE, alpha = 0.3, colour="deepskyblue") + 
     labs(title = "Quality metrics vs Demographic Variables") + 
     theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5))
}
x_var <- names(combined_final)[33]
y_var <- names(combined_final)[c(6,9:13)]
map2(x_var, y_var, create_scatter)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

[[6]]

Second: “Safety and Respect vs Demographic Variables” - colour: darkorange

create_scatter2 <- function(x,y) {
  ggplot(data = combined_final) + 
     aes_string(x = x, y = y) +  
     geom_point(na.rm=TRUE, alpha = 0.3, colour="darkorange") + 
     labs(title = "Safety and Respect vs Demographic Variables") + 
     theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5))
}
x_var <- names(combined_final)[29]
y_var <- names(combined_final)[c(6,9:13)]
map2(x_var, y_var, create_scatter2)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

[[6]]

Third: “Communication vs Demographic Variables” - colour: green2

create_scatter3 <- function(x,y) {
  ggplot(data = combined_final) + 
     aes_string(x = x, y = y) +  
     geom_point(na.rm=TRUE, alpha = 0.3, colour="green2") + 
     labs(title = "Communication vs Demographic Variables") + 
     theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5))
}
x_var <- names(combined_final)[30]
y_var <- names(combined_final)[c(6,9:13)]
map2(x_var, y_var, create_scatter3)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

[[6]]

Fourth: “Engagement vs Demographic Variables” - colour: blueviolet

create_scatter4 <- function(x,y) {
  ggplot(data = combined_final) + 
     aes_string(x = x, y = y) +  
     geom_point(na.rm=TRUE, alpha = 0.3, colour="blueviolet") + 
     labs(title = "Engagement vs Demographic Variables") + 
     theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5))
}
x_var <- names(combined_final)[31]
y_var <- names(combined_final)[c(6,9:13)]
map2(x_var, y_var, create_scatter4)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

[[6]]

Fourth: “Academic Expectations vs Demographic Variables” - colour: firebrick1

create_scatter5 <- function(x,y) {
  ggplot(data = combined_final) + 
     aes_string(x = x, y = y) +  
     geom_point(na.rm=TRUE, alpha = 0.3, colour="firebrick1") + 
     labs(title = "Academic Expectations vs Demographic Variables") + 
     theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5))
}
x_var <- names(combined_final)[32]
y_var <- names(combined_final)[c(6,9:13)]
map2(x_var, y_var, create_scatter5)

[[1]]

[[2]]

[[3]]

[[4]]

[[5]]

[[6]]

It would be interesting to see whether parents, students, and teachers have similar perceptions about the four school quality metrics they were surveyed about:

Safety and Respect
Communication
Engagement
Academic Expectations

We can group the data by question and survey response type (parent, student, teacher, or total) and then calculate a summary average for each group.

combined_survey_gather <- combined_final %>%
  gather(key = "survey_question", value = "score", 17:28,33) %>%
  mutate(response_type = str_split(`survey_question`, "_", simplify = TRUE)[,2], metric = str_split(`survey_question`, "_", simplify = TRUE)[,1]) 
  
combined_survey_gather <- combined_survey_gather %>%
  mutate(response_type = if_else(response_type == "p", "parent", if_else(response_type == "t", "teacher", if_else(response_type == "s", "student", "stp_combined"))))
  

Now its time to group the data into categories: Students, Teachers and Parents.

1. Summary perceptions:

perceptions <- combined_survey_gather %>%
  group_by(response_type, metric) %>%
  summarize(avg_score = mean(score, na.rm = TRUE))
perceptions

combined_survey_gather <- combined_survey_gather %>%
  drop_na(score)

combined_survey_gather %>%
 # filter(response_type != "quality") %>%
  ggplot() +
  aes(x = survey_question, y = score, fill = response_type) +
  geom_boxplot() + 
  labs(title = "Perceptions of quality education in NYC High Schools") + 
  theme(panel.background = element_rect(fill = "white"), plot.title = element_text(hjust = 0.5), axis.text.x=element_text(angle=90, hjust=1, size=8), panel.spacing.x=unit(0.5, "lines"))  

There are still interesting questions we can answer:

• Is there a relationship between gender percentages and average SAT scores? How about for the different sections of the SAT (Reading, Writing, and Math)?

• Which NYC schools seem to have the best quality metrics according to survey data? Is there a difference if we break this down by response type?

• Can we learn anything from the differences in school quality metric perception between groups? For example, if we created a new variable by subtracting saf_p_11 from saf_p_11, can this difference in how students and parents perceive safety may be related to any other variables?